{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "[Introduction](introyt1_tutorial.html) ||\n",
        "[Tensors](tensors_deeper_tutorial.html) ||\n",
        "[Autograd](autogradyt_tutorial.html) ||\n",
        "[Building Models](modelsyt_tutorial.html) ||\n",
        "**TensorBoard Support** ||\n",
        "[Training Models](trainingyt.html) ||\n",
        "[Model Understanding](captumyt.html)\n",
        "\n",
        "# PyTorch TensorBoard Support\n",
        "\n",
        "Follow along with the video below or on [youtube](https://www.youtube.com/watch?v=6CEld3hZgqc)_.\n",
        "\n",
        ".. raw:: html\n",
        "\n",
        "   <div style=\"margin-top:10px; margin-bottom:10px;\">\n",
        "     <iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/6CEld3hZgqc\" frameborder=\"0\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n",
        "   </div>\n",
        "\n",
        "## Before You Start\n",
        "\n",
        "To run this tutorial, you’ll need to install PyTorch, TorchVision,\n",
        "Matplotlib, and TensorBoard.\n",
        "\n",
        "With ``conda``:\n",
        "\n",
        "``conda install pytorch torchvision -c pytorch``\n",
        "``conda install matplotlib tensorboard``\n",
        "\n",
        "With ``pip``:\n",
        "\n",
        "``pip install torch torchvision matplotlib tensorboard``\n",
        "\n",
        "Once the dependencies are installed, restart this notebook in the Python\n",
        "environment where you installed them.\n",
        "\n",
        "\n",
        "## Introduction\n",
        " \n",
        "In this notebook, we’ll be training a variant of LeNet-5 against the\n",
        "Fashion-MNIST dataset. Fashion-MNIST is a set of image tiles depicting\n",
        "various garments, with ten class labels indicating the type of garment\n",
        "depicted. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# PyTorch model and training necessities\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "\n",
        "# Image datasets and image manipulation\n",
        "import torchvision\n",
        "import torchvision.transforms as transforms\n",
        "\n",
        "# Image display\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "# PyTorch TensorBoard support\n",
        "from torch.utils.tensorboard import SummaryWriter"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Showing Images in TensorBoard\n",
        "\n",
        "Let’s start by adding sample images from our dataset to TensorBoard:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiYAAACxCAYAAADwMnaUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAmBklEQVR4nO3de3BU5fkH8G8CSQiXLISYhBACQdEgtyKBGHFaq1FkGISCVhE1VWYcNSiXWjUqOLXQoLZeUITWaaG1IkorWKjg0ABhsFzDRRAIqEgCIeGahFsuJuf3R8v+eL+77skmG/aEfD8zmfHZc/bs2fdceN33Oc8bYlmWBREREREHCA32DoiIiIhcpI6JiIiIOIY6JiIiIuIY6piIiIiIY6hjIiIiIo6hjomIiIg4hjomIiIi4hjqmIiIiIhjqGMiIiIijqGOiYiIiDhGk3VM5syZgx49eqBNmzZIS0vD5s2bm+qjRERE5AoR0hRz5Xz00Ud46KGHMG/ePKSlpeHNN9/E4sWLUVBQgNjYWJ/vraurQ3FxMTp06ICQkJBA75qIiIg0AcuycObMGSQkJCA0tOG/ezRJxyQtLQ2DBw/GO++8A+C/nY1u3brhySefxHPPPefzvYcPH0a3bt0CvUsiIiJyGRQVFSExMbHB728dwH0BAFRXVyM/Px/Z2dnu10JDQ5GRkYENGzZ4rF9VVYWqqip3fLGfNGPGDLRp0ybQuyciIiJNoLKyEi+++CI6dOjQqO0EvGNy4sQJ1NbWIi4uzng9Li4O+/bt81g/JycHv/71rz1eb9OmDSIjIwO9eyIiItKEGpuGEfSncrKzs1FeXu7+KyoqCvYuiYiISJAE/BeTmJgYtGrVCqWlpcbrpaWliI+P91g/IiICERERgd4NERERaYYC/otJeHg4Bg0ahNzcXPdrdXV1yM3NRXp6eqA/TkRERK4gAf/FBACmTp2KzMxMpKamYsiQIXjzzTdx7tw5PPzww03xcSIiInKFaJKOyb333ovjx49j+vTpKCkpwY9+9COsXLnSIyG2oZ544omAbCeQZsyYYcTh4eFGXFNT43N5SUmJEVdWVhpx9+7djbi6utqIW7VqZcQHDhww4rlz5xqxE4bP3n33XZ/LnXCc7Z6mt0vy4oTvS39JBDzPg0ufUAPg0Zlv166dz8+z4+37NHW9oOZwnKXxnHacG1IJg6+Fb775xogzMjJ8rl9XV2fEXMvj6aefNmK7NuHt8ecFo9aX3XEOhCbpmADAxIkTMXHixKbavIiIiFyBgv5UjoiIiMhF6piIiIiIYzTZUM6Vjscvp02bZsSdO3c2Ys4J4VyBzz77zIg59+C1114z4k8//dSIuUpuRUWFET/zzDNGnJKSAvFkN0ZsZ9euXUY8evRoI27btq0Rh4WFGfGpU6eMeMqUKUbM55G/Y8zByDERaQp8LjdF/sUDDzxgxCdOnDBivu/W1tb63IesrCwj5vtDQkKCEft7/7lSrm/9YiIiIiKOoY6JiIiIOIY6JiIiIuIYyjFpIC65z+N4sbGxRnzhwgUj5twCrvHy3XffGfE//vEPn+vzbI481sh1TZRj4p2/Y7orV6404rfeesuIOVeIuVwuI+bzhKcO5+rJnJsUHR3t8/P8/X4iTtWQ3ImvvvrKiF966SUj3rhxoxFzPanrrrvOiK+55hoj5rpFvI98n+7fv78R9+jRw4jff/99I+7duzd8aY75JN7oLiUiIiKOoY6JiIiIOIY6JiIiIuIYyjFpoOXLl/tcznPXHDp0yIhnzpxpxC+88IIRc27BvHnzjDgzM9OIeeyTcxdiYmJ87q/81/nz54149uzZRrx+/XojtssJOXr0qBHzGPAdd9xhxAsWLDBizkXq2rWrEY8dO9aIe/XqZcS33nqrEY8YMQKMx71FmqM33njDiP/85z97rFNQUGDEXFeI60/xfbSsrMyIOdfwlltuMeKPP/7YiDt16uTz84uKiox40KBBRjxw4EAjHj58uBG/+OKLsGNX/8UJ9IuJiIiIOIY6JiIiIuIY6piIiIiIY6hjIiIiIo6h5NcGOnPmjBHzZE6cUNS9e3cjnjFjhhFzEmXfvn2NeMeOHUbcrVs3I+bk2lmzZhnxVVddBbHHSclbtmwxYk4U5eRULqjGk3Jt377diBcvXmzEnCzbp08fI+bzpHVr8xLmQnpffvmlEe/fvx9s+vTpHq+JBJtdkubOnTuN+Je//KURd+nSxWObV199tRFHRUUZMU+SyddXZGSkEXfs2NGIT58+bcSc7Mrf6fvvvzdifkiBC3XyfZ4LxN1+++1gaWlpPvdBya8iIiIiPqhjIiIiIo6hjomIiIg4hnJMGojH+jj3oLa21oh5bJIna+KxxpMnTxrx8ePHjZgLbZWUlBgxj42KJx5rBTxzPOLj442YjxNvg3NA+P08nltcXGzEPObNBdO4ANzZs2d9vr+qqsqIv/nmG4g0B3a5D6+88ooRc95dz549Pd7DBdL4Psk5Y3z98Pq8vbq6Op/b45ivZ/48zmHhvBnOaeNCnIBnjklzmMjT+XsoIiIiLYY6JiIiIuIY6piIiIiIYygRIUB47JAn8eOxQ2/5DZeKi4szYs5d4Oft+Xn5Dz/80Ii9Pd/e0h07dszjNa5Pw7lBjJfz+zn3iCdbXLt2rREPGzbMiPm84XF3rp/DY9bt27c3Ys5VAjzPJR63FnECPk83btxoxFyrqaamxmMbnLsXHR1txHzf5kn2OD+D94mXc06Kt326FF97nCezbds2I+Z/d3bt2uWxTc57UY6JiIiIiB/UMRERERHHUMdEREREHEM5Jg1kN07HY5U81sg5Jlz3pLKy0udyzjXguiZffPGFEfPcPPPnz/e22y1KYWGhx2ucy8NjvjwmzetznRMeU05MTDRizgHh3KJz584ZMddZueGGG4yY5+rg93MMeObFdO7c2WMdkWBbs2aNEfO5ztcm52MAnjlhHPP1XFFRYcScc2KXK8j43wHO/+BcQb7vf/vtt0Y8ZMgQIy4tLfX4TK65lZycXL+dDSL9YiIiIiKOoY6JiIiIOIbfHZN169Zh5MiRSEhIQEhICJYuXWostywL06dPR5cuXRAZGYmMjAyPqdhFREREvPE7x+TcuXMYMGAAHnnkEYwZM8Zj+auvvorZs2fjL3/5C5KTkzFt2jQMGzYMe/bs8ai50JzZjS1yjgnH/m6f3283j0RCQoIRd+/e3a/Pbwl27tzp8RqP+XLOCI/hci4Qj0HbLee6JTzGzecBX0MnTpwwYs4X4TFtrnPibRvKMfFfc6wVwecWfwd/71lN7bPPPjNivjZdLpcRe9t/zvmyq33y3XffGTHnaPH1yPvA1xvXNeLP47mudu/ebcSck8b/DnBOCgD861//MuKJEyd6rOM0fndMhg8fjuHDh3tdZlkW3nzzTbz44osYNWoUAOCvf/0r4uLisHTpUtx3332N21sRERG5ogW0W3/w4EGUlJQgIyPD/ZrL5UJaWho2bNjg9T1VVVWoqKgw/kRERKRlCmjHpKSkBIDnI49xcXHuZSwnJwcul8v9x1NXi4iISMsR9Dom2dnZmDp1qjuuqKhoFp0THgvk8U4er7WrY8L1MjhXgMes7cZ/eeyR51wRz3kzAM925RwRHlPmTjjPneGtbsil+Lywq3/DtRpiYmKMuKyszOfneVNQUGDEPJ+PeI7d83Gyyynh3CQ+L/gBAc49Yv7Wz/D2Hn/vKcHGtZk4n4PvwXwtAp5twHWBuI4I54TxceP7Nrch55jw/aNdu3Y+38/XM9dd4f3h+wUA5Ofne7zmdAH9xSQ+Ph6A50VYWlrqXsYiIiIQFRVl/ImIiEjLFNCOSXJyMuLj45Gbm+t+raKiAps2bUJ6enogP0pERESuQH4P5Zw9exZff/21Oz548CB27NiB6OhoJCUlYfLkyZgxYwZ69erlflw4ISEBo0ePDuR+i4iIyBXI747J1q1b8dOf/tQdX8wPyczMxIIFC/DMM8/g3LlzePTRR1FWVoabb74ZK1euvKJqmACeY4/8PDqP39rNdcM5JVxbIjY21uf2eWyS94dzE8T7GDS3K6/Dx5HHmHl5WFiYEfNx4uPKY8qcG8S5S5ybwOcVj7t7yyO49H80WirOPbCrI8R4DpNdu3YZ8ebNm42Ya0C98847RsxD2vyLs10dI2/s3sPn+v3332/E3AYfffSR3/vQGEVFRUbMNUH4WvSWh8PrnD171ojLy8uN2C53kNuMc0q4zTgnhHNa+PN4ff7O/O8G57wBwPbt2z1eczq/Oya33HKLz8SrkJAQvPzyy3j55ZcbtWMiIiLS8ji/PKGIiIi0GOqYiIiIiGMEvY5Jc8W5A5zTweO5ds+nJyUl+fw8Hrvk59kZj4XyM//imccDeB5HHpPmOgG8PuPjzI/N8z5w3RS7HBYec+a5dngM3Nsw7JEjRzxeu5J4+858fdrFXAuiuLjYiI8dO2bEycnJRjxixAgjfv31142Yr+eZM2f63J9x48YZ8eDBg42Yr3/A81z88MMPjfjLL7804tOnTxsxn2t2NXoCjfOteE4nzs/gGPhveQpf63A78/XI93E+t/h+YLeccakNvt9wTgnnbnqrY8LHtTnQLyYiIiLiGOqYiIiIiGOoYyIiIiKOoRyTBuKcEB7/5LFIjnkscdKkSUbMY5vPP/+8EXfp0sWIeWyRcwt4PFaAw4cPe7zGdUx4TJjHqHksn99vl1PCuQV2dVSYXW4En0ecGwV45ksEG4/LN6Rmh7/v51yD+fPnG/H+/fuNuFevXkZ86YzqAPD+++8bMc+u3rFjRyPm4865A1xf46mnnjJiPg85P8TbNrk2Ci/v2rWrEXNOibfrJ5D43OWaI5x/xXM+ecP1p/i4c/0YPhftrnfO8eKcMMb/bvBx5nOXcw35/RUVFR6f0ZB5lYJNv5iIiIiIY6hjIiIiIo6hjomIiIg4hnJMGoif6eexRrtxPbuxR84t4LFNu3FzHjvVXDmevI3HtmvXzoh5nJvbnY8LH3eeC4ePq11Oi91x55wRnk+E959zGwDgu+++83gtmBqbU8J43hoAWLdunRFzLlDfvn2NuFu3bkbM8ws9+OCDRsw5X1dddZUR83FhnMvUs2dPIx4wYIAR83nCuRKA57nCORtcz4bzVPie1dTnDedbeKvNcqn65FJwDgjH/Bl8H+UcFbucFWaXQ2Z3vfN35Lwgu/MK8KyFwv+WOYF+MRERERHHUMdEREREHEMdExEREXEM5ZgECI8N8ngsj+/y2GJCQoIR81ih3fgpj23y+t5yC1o6b2PWfFy4TgAfR17O7c7zk3DuAR83rnfD7HJceH+41oO3OiZOmyuH24SPCddy4NowX3zxhRF7m4dq1KhRRszX6/bt242Y65hwHaKbb77ZiDk/g78DH0fOLeLcB56L59SpU0Zsd14Cnu1mVyeE24TzGXifAo2vHcbfmeOG5JzYXX9c24VrAHG+Bp/L3q6/S/k7FxfXp/I2/xfj6z0lJcX2PZebfjERERERx1DHRERERBxDHRMRERFxDHVMRERExDGU/NpAXIiLE63skqg4adHlchkxJ8PaJXLxck664u0L0L59e4/XOOGPjxMfd8ZJjtzuPBEaJ0Fy0iIv52Q9LtzFhcA4SdPbecTbuNw4kXPChAlG3KdPHyOOiYkx4sLCQiNOTk42Ym/HjCfVKykpMWJOBOXrmZPJT548acR8HDmRk5NjebldoT27Ao3ekib52HO7cPEvTuTcunWrET/wwAM+96GxuAAi7z8nfvIx8Ia/I98DOCG4U6dORsztzueJ3bXE1zMnp/Nxtiu4xgUV+ft5w0nLSn4VERER8UEdExEREXEMdUxERETEMZRj0kBcKIfHHjnmSbkYj+dyrgLjMWQeA+fPd+JETcHG+ReA5xgx5xJERkYaMY9rc4EkPg68nMeQefveCmVd6vjx40bcu3dvI+bvaJebEAw84dyCBQuM+OjRo0a8fv16I+Y25Un7OJ8E8GwXzlfg48J5MJzDxbkKPNbP+Rw33nijEXNeTPfu3Y3Y7jy0y1Xwtg9cMI1jvmdxzO3+7rvvenxmY9gVWOPzpj6TP/I+c5vwBHecs3Xo0CEj5uNgl+PBeTJ83PjzefuM7z/1yRfjyRGdSL+YiIiIiGOoYyIiIiKOoY6JiIiIOIZyTALEbmzRbrI3HkPmOgfM7vl23r63mh3iiXN1rr32WiPmXJ2CggIj5rolXDOAj4tdDord/vznP/8xYj5v7HIRvO3D5cZ5Mpxb0KNHDyMORN0FPg7ecjIuxddXfepFOB1/Z7s24fOEa+oEGtcUsav5w3k/3s51vg9ybDcRIO8T5+XYnUd83vD6nFuYmJjoc/nevXuNmO8f3hw+fNh2nWBr/leXiIiIXDH86pjk5ORg8ODB6NChA2JjYzF69GiP/2OsrKxEVlYWOnfujPbt22Ps2LFen34QERERYX51TPLy8pCVlYWNGzdi1apVqKmpwR133GGU2Z4yZQqWLVuGxYsXIy8vD8XFxRgzZkzAd1xERESuPH7lmKxcudKIFyxYgNjYWOTn5+PHP/4xysvL8ac//QkLFy7ErbfeCgCYP38+evfujY0bN3o8u9+ccS6B3dii3XKek4HnQLDD4788lmlXR6Ul4PFhb7kVPG595513GvGKFSt8boPbmT+Tc0Z4DJvn3mAnTpzwuT2u7cD5GFyHAfCsnXC5denSxYj5O/AvrpzbwLkHdvU6AM924+vFbjnjukJ2+Rp2cyLZ5YzZzZ3ljV2ejF3eml0bBdr58+d9fj6fy5yrlJqa6rFNbne+XvncsWsDzknh5XbHiXNc+DtwTondfbw+x4TnIHKiRuWYXCzUEh0dDQDIz89HTU0NMjIy3OukpKQgKSnJa5EjERERkUs1+Kmcuro6TJ48GUOHDkXfvn0B/HeGzvDwcI8nTOLi4jxm77yoqqrK6MU2h96ciIiINI0G/2KSlZWF3bt3Y9GiRY3agZycHLhcLvcflwAWERGRlqNBv5hMnDgRy5cvx7p164znrOPj41FdXY2ysjLjV5PS0lLEx8d73VZ2djamTp3qjisqKppF54RzTPyti8C43oTdvA922/d3DoeWgJ/fv+aaa2zfw2POBw4cMGI+D5jduDznqPBx49oMnA/CdVV4+3zdff311x77yPO0BBu3OceM8zv42uA8H8A+Z8Pbey5llytkN2eS3dxYdu9nvNxbrgHfU/g9XPeD1+dz0e64NJZdPghfC5x71KtXL49t5ufnGzHneHAb8Llldxx4H+3u07GxsUbMIwvc5mlpaUa8Y8cOI67PcedtOpFf/1pZloWJEydiyZIlWL16tccNbdCgQQgLC0Nubq77tYKCAhQWFiI9Pd3rNiMiIhAVFWX8iYiISMvk1y8mWVlZWLhwIT799FN06NDB3btzuVyIjIyEy+XChAkTMHXqVERHRyMqKgpPPvkk0tPTr6gnckRERKRp+NUxmTt3LgDglltuMV6fP38+fvGLXwAA3njjDYSGhmLs2LGoqqrCsGHDAj4dtoiIiFyZ/OqY1OfZ+TZt2mDOnDmYM2dOg3eqObB7npzHJgP9zL/dPBecqyDAqVOnjNhbHg8f148//tiIeVyba3Bw7QXOQbn4aP1F+/fvN2K7+hl2x3nnzp1GzE/I8ecDnk/CXVowEWj6XILG4loSTDV8mifOheBrye6eyvkYADzKVvB9mlMJOM/FLvfHLqfEbm4e/o4899UNN9xgxAsWLPC5PcA+t8iJlBEpIiIijqGOiYiIiDiGOiYiIiLiGA2u/NrS2Y1r2807wXi8lMcy7djVwxBg8+bNRsw5J4BnPsV1111nxDwGzcetbdu2PpfzGDOPKfO4Oo+B83nH51VcXJwR9+/f34iXLl0Kxnkzu3fvNmJvY/UiTY3voVz7xS736dprr/V4jfOnOIeLc7K49pFdToldLRj+PK5fw3Om8TxRAwcOhC/e5oXidrP7t8sJ9IuJiIiIOIY6JiIiIuIY6piIiIiIYyjHpIHsnqG3q0fBuNaCt7FCX9vj+hbNYRzxchs5cqQR33PPPR7rcN0CrkuyevVqn+vzcSwvLzdizms5e/asEfNx5BwTXs7zg/D677//vhFzcUTAc1zd6XVLpGXgPLvjx48bsd091dtcWJwjwjlkPJcU54xwDghfK3y/4LpBnHPC1zOvzzkoCQkJ8FdzrOOjX0xERETEMdQxEREREcdQx0REREQcQzkmDcTjn42dj4BzUvytQ+LvnA0tEY8Hx8TE+L2Nv//970a8d+9eIz5y5IgRX5yB+6IePXoY8cGDB42YzyOuaxIbG2vEXbt2NWK7OgeDBw/2uRwANm3aZMQNGdcWaSyu+cPnOtckYd5ypfh63LdvnxEXFBQYMed4VFZWGjHnmHEdI7ucNb7+ueZIcXGxEZ84ccKI+R7G94vmSr+YiIiIiGOoYyIiIiKOoY6JiIiIOIZyTBqIxzc5p8PfOiJc74LHIhmPffLnnz592q/PbwkqKiqMuCE5JjxXzpU4j4zduSdyOXC+hN18Ypxf5a3OCedwcB0hrmvCOSV8n+XP4HsK1xni78Bz8XAO2k033WTEPXv2NGKus+JtjjXOS7Obt80J9IuJiIiIOIY6JiIiIuIY6piIiIiIY6hjIiIiIo6h5NcG4iTIfv36GTEnp9pNysfJst98840Ru1wuI+akJ04U69u3r8/Pk4bh5De7ZLhAJ5rxpF92n2830Zk3dueqyOXAk0vyZHScuMoT7PGEeIBnAnxzx9/RW/IrF33r06dPk+5TIOgXExEREXEMdUxERETEMdQxEREREcdQjkkDcaGcXbt2GTGPd/L4KONxQC60wwXYeDKnkydPGnG3bt18fl5LVJ+JDTmHg3NEApHD0Rh2+8Psvg/g2S5lZWUN2zmRAMrNzTViLkZmN6FdS8B5NsePH/dYhydDXLt2rRGPHDky4PvVWPrFRERERBxDHRMRERFxDHVMRERExDGUYxIg7733nhEXFhYaMU8OVVpa6nN7qampRvz888/7XJ/zBO6++26f67dELbE+R33qqHCeyjXXXNNUuyNSb/fff78Rc94c50IFYkJNzslqLH/rGNUnJ+xS2dnZRuxt8tYuXboYcf/+/f3ap2DQLyYiIiLiGH51TObOnYv+/fsjKioKUVFRSE9Px4oVK9zLKysrkZWVhc6dO6N9+/YYO3as7S8DIiIiIhf51TFJTEzErFmzkJ+fj61bt+LWW2/FqFGj8NVXXwEApkyZgmXLlmHx4sXIy8tDcXExxowZ0yQ7LiIiIleeEKuRg2rR0dF47bXXcPfdd+Oqq67CwoUL3fkN+/btQ+/evbFhwwbceOON9dpeRUUFXC4Xfve73yEyMrIxuyYiIiKXyYULF/D000+jvLzcYz45fzQ4x6S2thaLFi3CuXPnkJ6ejvz8fNTU1CAjI8O9TkpKCpKSkrBhw4Yf3E5VVRUqKiqMPxEREWmZ/O6Y7Nq1C+3bt0dERAQee+wxLFmyBNdffz1KSkoQHh7uMSNkXFwcSkpKfnB7OTk5cLlc7j9VLBUREWm5/O6YXHfdddixYwc2bdqExx9/HJmZmdizZ0+DdyA7Oxvl5eXuv6KiogZvS0RERJo3v+uYhIeHu+scDBo0CFu2bMFbb72Fe++9F9XV1SgrKzN+NSktLUV8fPwPbi8iIsJ2HhkRERFpGRpdx6Surg5VVVUYNGgQwsLCjImXCgoKUFhYiPT09MZ+jIiIiLQAfv1ikp2djeHDhyMpKQlnzpzBwoULsXbtWnz++edwuVyYMGECpk6diujoaERFReHJJ59Eenp6vZ/IERERkZbNr47JsWPH8NBDD+Ho0aNwuVzo378/Pv/8c9x+++0AgDfeeAOhoaEYO3YsqqqqMGzYMLz77rt+7dDFp5crKyv9ep+IiIgEz8V/txtb2r/RdUwC7fDhw3oyR0REpJkqKipCYmJig9/vuI5JXV0diouLYVkWkpKSUFRU1KhCLS1dRUUFunXrpnZsBLVh46kNA0Pt2Hhqw8b7oTa0LAtnzpxBQkKCx+Sg/nDc7MKhoaFITEx0F1q7OC+PNI7asfHUho2nNgwMtWPjqQ0bz1sbulyuRm9XswuLiIiIY6hjIiIiIo7h2I5JREQEXnrpJRVfayS1Y+OpDRtPbRgYasfGUxs2XlO3oeOSX0VERKTlcuwvJiIiItLyqGMiIiIijqGOiYiIiDiGOiYiIiLiGI7tmMyZMwc9evRAmzZtkJaWhs2bNwd7lxwrJycHgwcPRocOHRAbG4vRo0ejoKDAWKeyshJZWVno3Lkz2rdvj7Fjx6K0tDRIe+x8s2bNQkhICCZPnux+TW1YP0eOHMEDDzyAzp07IzIyEv369cPWrVvdyy3LwvTp09GlSxdERkYiIyMDBw4cCOIeO0ttbS2mTZuG5ORkREZG4uqrr8ZvfvMbY/4RtaFp3bp1GDlyJBISEhASEoKlS5cay+vTXqdOncL48eMRFRWFjh07YsKECTh79uxl/BbB56sda2pq8Oyzz6Jfv35o164dEhIS8NBDD6G4uNjYRiDa0ZEdk48++ghTp07FSy+9hG3btmHAgAEYNmwYjh07Fuxdc6S8vDxkZWVh48aNWLVqFWpqanDHHXfg3Llz7nWmTJmCZcuWYfHixcjLy0NxcTHGjBkTxL12ri1btuAPf/gD+vfvb7yuNrR3+vRpDB06FGFhYVixYgX27NmD3//+9+jUqZN7nVdffRWzZ8/GvHnzsGnTJrRr1w7Dhg3TxJ3/88orr2Du3Ll45513sHfvXrzyyit49dVX8fbbb7vXURuazp07hwEDBmDOnDlel9envcaPH4+vvvoKq1atwvLly7Fu3To8+uijl+srOIKvdjx//jy2bduGadOmYdu2bfjkk09QUFCAu+66y1gvIO1oOdCQIUOsrKwsd1xbW2slJCRYOTk5Qdyr5uPYsWMWACsvL8+yLMsqKyuzwsLCrMWLF7vX2bt3rwXA2rBhQ7B205HOnDlj9erVy1q1apX1k5/8xJo0aZJlWWrD+nr22Wetm2+++QeX19XVWfHx8dZrr73mfq2srMyKiIiwPvzww8uxi443YsQI65FHHjFeGzNmjDV+/HjLstSGdgBYS5Ysccf1aa89e/ZYAKwtW7a411mxYoUVEhJiHTly5LLtu5NwO3qzefNmC4B16NAhy7IC146O+8Wkuroa+fn5yMjIcL8WGhqKjIwMbNiwIYh71nyUl5cDAKKjowEA+fn5qKmpMdo0JSUFSUlJalOSlZWFESNGGG0FqA3r65///CdSU1Nxzz33IDY2FgMHDsR7773nXn7w4EGUlJQY7ehyuZCWlqZ2/J+bbroJubm52L9/PwBg586dWL9+PYYPHw5Abeiv+rTXhg0b0LFjR6SmprrXycjIQGhoKDZt2nTZ97m5KC8vR0hICDp27AggcO3ouEn8Tpw4gdraWsTFxRmvx8XFYd++fUHaq+ajrq4OkydPxtChQ9G3b18AQElJCcLDw90nz0VxcXEoKSkJwl4606JFi7Bt2zZs2bLFY5nasH6+/fZbzJ07F1OnTsXzzz+PLVu24KmnnkJ4eDgyMzPdbeXt+lY7/tdzzz2HiooKpKSkoFWrVqitrcXMmTMxfvx4AFAb+qk+7VVSUoLY2FhjeevWrREdHa02/QGVlZV49tlnMW7cOPdEfoFqR8d1TKRxsrKysHv3bqxfvz7Yu9KsFBUVYdKkSVi1ahXatGkT7N1pturq6pCamorf/va3AICBAwdi9+7dmDdvHjIzM4O8d83Dxx9/jA8++AALFy5Enz59sGPHDkyePBkJCQlqQ3GEmpoa/PznP4dlWZg7d27At++4oZyYmBi0atXK42mH0tJSxMfHB2mvmoeJEydi+fLlWLNmDRITE92vx8fHo7q6GmVlZcb6atP/l5+fj2PHjuGGG25A69at0bp1a+Tl5WH27Nlo3bo14uLi1Ib10KVLF1x//fXGa71790ZhYSEAuNtK1/cP+9WvfoXnnnsO9913H/r164cHH3wQU6ZMQU5ODgC1ob/q017x8fEeD1d8//33OHXqlNqUXOyUHDp0CKtWrXL/WgIErh0d1zEJDw/HoEGDkJub636trq4Oubm5SE9PD+KeOZdlWZg4cSKWLFmC1atXIzk52Vg+aNAghIWFGW1aUFCAwsJCten/3Hbbbdi1axd27Njh/ktNTcX48ePd/602tDd06FCPR9X379+P7t27AwCSk5MRHx9vtGNFRQU2bdqkdvyf8+fPIzTUvDW3atUKdXV1ANSG/qpPe6Wnp6OsrAz5+fnudVavXo26ujqkpaVd9n12qoudkgMHDuDf//43OnfubCwPWDs2IFm3yS1atMiKiIiwFixYYO3Zs8d69NFHrY4dO1olJSXB3jVHevzxxy2Xy2WtXbvWOnr0qPvv/Pnz7nUee+wxKykpyVq9erW1detWKz093UpPTw/iXjvfpU/lWJbasD42b95stW7d2po5c6Z14MAB64MPPrDatm1r/e1vf3OvM2vWLKtjx47Wp59+an355ZfWqFGjrOTkZOvChQtB3HPnyMzMtLp27WotX77cOnjwoPXJJ59YMTEx1jPPPONeR21oOnPmjLV9+3Zr+/btFgDr9ddft7Zv3+5+WqQ+7XXnnXdaAwcOtDZt2mStX7/e6tWrlzVu3LhgfaWg8NWO1dXV1l133WUlJiZaO3bsMP6tqaqqcm8jEO3oyI6JZVnW22+/bSUlJVnh4eHWkCFDrI0bNwZ7lxwLgNe/+fPnu9e5cOGC9cQTT1idOnWy2rZta/3sZz+zjh49Grydbga4Y6I2rJ9ly5ZZffv2tSIiIqyUlBTrj3/8o7G8rq7OmjZtmhUXF2dFRERYt912m1VQUBCkvXWeiooKa9KkSVZSUpLVpk0bq2fPntYLL7xg3PzVhqY1a9Z4vQdmZmZallW/9jp58qQ1btw4q3379lZUVJT18MMPW2fOnAnCtwkeX+148ODBH/y3Zs2aNe5tBKIdQyzrknKCIiIiIkHkuBwTERERabnUMRERERHHUMdEREREHEMdExEREXEMdUxERETEMdQxEREREcdQx0REREQcQx0TERERcQx1TERERMQx1DERERERx1DHRERERBxDHRMRERFxjP8DRmAtJ+KO5toAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Gather datasets and prepare them for consumption\n",
        "transform = transforms.Compose(\n",
        "    [transforms.ToTensor(),\n",
        "    transforms.Normalize((0.5,), (0.5,))])\n",
        "\n",
        "# Store separate training and validations splits in ./data\n",
        "training_set = torchvision.datasets.FashionMNIST('./data',\n",
        "    download=True,\n",
        "    train=True,\n",
        "    transform=transform)\n",
        "validation_set = torchvision.datasets.FashionMNIST('./data',\n",
        "    download=True,\n",
        "    train=False,\n",
        "    transform=transform)\n",
        "\n",
        "training_loader = torch.utils.data.DataLoader(training_set,\n",
        "                                              batch_size=4,\n",
        "                                              shuffle=True,\n",
        "                                              num_workers=2)\n",
        "\n",
        "\n",
        "validation_loader = torch.utils.data.DataLoader(validation_set,\n",
        "                                                batch_size=4,\n",
        "                                                shuffle=False,\n",
        "                                                num_workers=2)\n",
        "\n",
        "# Class labels\n",
        "classes = ('T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',\n",
        "        'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle Boot')\n",
        "\n",
        "# Helper function for inline image display\n",
        "def matplotlib_imshow(img, one_channel=False):\n",
        "    if one_channel:\n",
        "        img = img.mean(dim=0)\n",
        "    img = img / 2 + 0.5     # unnormalize\n",
        "    npimg = img.numpy()\n",
        "    if one_channel:\n",
        "        plt.imshow(npimg, cmap=\"Greys\")\n",
        "    else:\n",
        "        plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
        "\n",
        "# Extract a batch of 4 images\n",
        "dataiter = iter(training_loader)\n",
        "images, labels = next(dataiter)\n",
        "\n",
        "# Create a grid from the images and show them\n",
        "img_grid = torchvision.utils.make_grid(images)\n",
        "matplotlib_imshow(img_grid, one_channel=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Above, we used TorchVision and Matplotlib to create a visual grid of a\n",
        "minibatch of our input data. Below, we use the ``add_image()`` call on\n",
        "``SummaryWriter`` to log the image for consumption by TensorBoard, and\n",
        "we also call ``flush()`` to make sure it’s written to disk right away.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Default log_dir argument is \"runs\" - but it's good to be specific\n",
        "# torch.utils.tensorboard.SummaryWriter is imported above\n",
        "writer = SummaryWriter('runs/fashion_mnist_experiment_1')\n",
        "\n",
        "# Write image data to TensorBoard log dir\n",
        "writer.add_image('Four Fashion-MNIST Images', img_grid)\n",
        "writer.flush()\n",
        "\n",
        "# To view, start TensorBoard on the command line with:\n",
        "#   tensorboard --logdir=runs\n",
        "# ...and open a browser tab to http://localhost:6006/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you start TensorBoard at the command line and open it in a new\n",
        "browser tab (usually at [localhost:6006](localhost:6006)_), you should\n",
        "see the image grid under the IMAGES tab.\n",
        "\n",
        "## Graphing Scalars to Visualize Training\n",
        "\n",
        "TensorBoard is useful for tracking the progress and efficacy of your\n",
        "training. Below, we’ll run a training loop, track some metrics, and save\n",
        "the data for TensorBoard’s consumption.\n",
        "\n",
        "Let’s define a model to categorize our image tiles, and an optimizer and\n",
        "loss function for training:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "class Net(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Net, self).__init__()\n",
        "        self.conv1 = nn.Conv2d(1, 6, 5)\n",
        "        self.pool = nn.MaxPool2d(2, 2)\n",
        "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
        "        self.fc1 = nn.Linear(16 * 4 * 4, 120)\n",
        "        self.fc2 = nn.Linear(120, 84)\n",
        "        self.fc3 = nn.Linear(84, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.pool(F.relu(self.conv1(x)))\n",
        "        x = self.pool(F.relu(self.conv2(x)))\n",
        "        x = x.view(-1, 16 * 4 * 4)\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = F.relu(self.fc2(x))\n",
        "        x = self.fc3(x)\n",
        "        return x\n",
        "    \n",
        "\n",
        "net = Net()\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now let’s train a single epoch, and evaluate the training vs. validation\n",
        "set losses every 1000 batches:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2500\n",
            "Batch 1000\n",
            "Batch 2000\n",
            "Batch 3000\n",
            "Batch 4000\n",
            "Batch 5000\n",
            "Batch 6000\n",
            "Batch 7000\n",
            "Batch 8000\n",
            "Batch 9000\n",
            "Batch 10000\n",
            "Batch 11000\n",
            "Batch 12000\n",
            "Batch 13000\n",
            "Batch 14000\n",
            "Batch 15000\n",
            "Finished Training\n"
          ]
        }
      ],
      "source": [
        "print(len(validation_loader))\n",
        "for epoch in range(1):  # loop over the dataset multiple times\n",
        "    running_loss = 0.0\n",
        "\n",
        "    for i, data in enumerate(training_loader, 0):\n",
        "        # basic training loop\n",
        "        inputs, labels = data\n",
        "        optimizer.zero_grad()\n",
        "        outputs = net(inputs)\n",
        "        loss = criterion(outputs, labels)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "        running_loss += loss.item()\n",
        "        if i % 1000 == 999:    # Every 1000 mini-batches...\n",
        "            print('Batch {}'.format(i + 1))\n",
        "            # Check against the validation set\n",
        "            running_vloss = 0.0\n",
        "            \n",
        "            net.train(False) # Don't need to track gradents for validation\n",
        "            for j, vdata in enumerate(validation_loader, 0):\n",
        "                vinputs, vlabels = vdata\n",
        "                voutputs = net(vinputs)\n",
        "                vloss = criterion(voutputs, vlabels)\n",
        "                running_vloss += vloss.item()\n",
        "            net.train(True) # Turn gradients back on for training\n",
        "            \n",
        "            avg_loss = running_loss / 1000\n",
        "            avg_vloss = running_vloss / len(validation_loader)\n",
        "            \n",
        "            # Log the running loss averaged per batch\n",
        "            writer.add_scalars('Training vs. Validation Loss',\n",
        "                            { 'Training' : avg_loss, 'Validation' : avg_vloss },\n",
        "                            epoch * len(training_loader) + i)\n",
        "\n",
        "            running_loss = 0.0\n",
        "print('Finished Training')\n",
        "\n",
        "writer.flush()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Switch to your open TensorBoard and have a look at the SCALARS tab.\n",
        "\n",
        "## Visualizing Your Model\n",
        "\n",
        "TensorBoard can also be used to examine the data flow within your model.\n",
        "To do this, call the ``add_graph()`` method with a model and sample\n",
        "input. When you open\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Again, grab a single mini-batch of images\n",
        "dataiter = iter(training_loader)\n",
        "images, labels = next(dataiter)\n",
        "\n",
        "# add_graph() will trace the sample input through your model,\n",
        "# and render it as a graph.\n",
        "writer.add_graph(net, images)\n",
        "writer.flush()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "When you switch over to TensorBoard, you should see a GRAPHS tab.\n",
        "Double-click the “NET” node to see the layers and data flow within your\n",
        "model.\n",
        "\n",
        "## Visualizing Your Dataset with Embeddings\n",
        "\n",
        "The 28-by-28 image tiles we’re using can be modeled as 784-dimensional\n",
        "vectors (28 \\* 28 = 784). It can be instructive to project this to a\n",
        "lower-dimensional representation. The ``add_embedding()`` method will\n",
        "project a set of data onto the three dimensions with highest variance,\n",
        "and display them as an interactive 3D chart. The ``add_embedding()``\n",
        "method does this automatically by projecting to the three dimensions\n",
        "with highest variance.\n",
        "\n",
        "Below, we’ll take a sample of our data, and generate such an embedding:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "collapsed": false
      },
      "outputs": [
        {
          "ename": "AttributeError",
          "evalue": "module 'tensorflow._api.v2.io.gfile' has no attribute 'get_filesystem'",
          "output_type": "error",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "Cell \u001b[1;32mIn [9], line 16\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[39m# log embeddings\u001b[39;00m\n\u001b[0;32m     15\u001b[0m features \u001b[39m=\u001b[39m images\u001b[39m.\u001b[39mview(\u001b[39m-\u001b[39m\u001b[39m1\u001b[39m, \u001b[39m28\u001b[39m \u001b[39m*\u001b[39m \u001b[39m28\u001b[39m)\n\u001b[1;32m---> 16\u001b[0m writer\u001b[39m.\u001b[39;49madd_embedding(features,\n\u001b[0;32m     17\u001b[0m                     metadata\u001b[39m=\u001b[39;49mclass_labels,\n\u001b[0;32m     18\u001b[0m                     label_img\u001b[39m=\u001b[39;49mimages\u001b[39m.\u001b[39;49munsqueeze(\u001b[39m1\u001b[39;49m))\n\u001b[0;32m     19\u001b[0m writer\u001b[39m.\u001b[39mflush()\n\u001b[0;32m     20\u001b[0m writer\u001b[39m.\u001b[39mclose()\n",
            "File \u001b[1;32md:\\Programs\\Python39\\lib\\site-packages\\torch\\utils\\tensorboard\\writer.py:926\u001b[0m, in \u001b[0;36mSummaryWriter.add_embedding\u001b[1;34m(self, mat, metadata, label_img, global_step, tag, metadata_header)\u001b[0m\n\u001b[0;32m    923\u001b[0m subdir \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39m%s\u001b[39;00m\u001b[39m/\u001b[39m\u001b[39m%s\u001b[39;00m\u001b[39m\"\u001b[39m \u001b[39m%\u001b[39m (\u001b[39mstr\u001b[39m(global_step)\u001b[39m.\u001b[39mzfill(\u001b[39m5\u001b[39m), \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_encode(tag))\n\u001b[0;32m    924\u001b[0m save_path \u001b[39m=\u001b[39m os\u001b[39m.\u001b[39mpath\u001b[39m.\u001b[39mjoin(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_get_file_writer()\u001b[39m.\u001b[39mget_logdir(), subdir)\n\u001b[1;32m--> 926\u001b[0m fs \u001b[39m=\u001b[39m tf\u001b[39m.\u001b[39;49mio\u001b[39m.\u001b[39;49mgfile\u001b[39m.\u001b[39;49mget_filesystem(save_path)\n\u001b[0;32m    927\u001b[0m \u001b[39mif\u001b[39;00m fs\u001b[39m.\u001b[39mexists(save_path):\n\u001b[0;32m    928\u001b[0m     \u001b[39mif\u001b[39;00m fs\u001b[39m.\u001b[39misdir(save_path):\n",
            "\u001b[1;31mAttributeError\u001b[0m: module 'tensorflow._api.v2.io.gfile' has no attribute 'get_filesystem'"
          ]
        }
      ],
      "source": [
        "# Select a random subset of data and corresponding labels\n",
        "def select_n_random(data, labels, n=100):\n",
        "    assert len(data) == len(labels)\n",
        "\n",
        "    perm = torch.randperm(len(data))\n",
        "    return data[perm][:n], labels[perm][:n]\n",
        "\n",
        "# Extract a random subset of data\n",
        "images, labels = select_n_random(training_set.data, training_set.targets)\n",
        "\n",
        "# get the class labels for each image\n",
        "class_labels = [classes[label] for label in labels]\n",
        "\n",
        "# log embeddings\n",
        "features = images.view(-1, 28 * 28)\n",
        "writer.add_embedding(features,\n",
        "                    metadata=class_labels,\n",
        "                    label_img=images.unsqueeze(1))\n",
        "writer.flush()\n",
        "writer.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now if you switch to TensorBoard and select the PROJECTOR tab, you\n",
        "should see a 3D representation of the projection. You can rotate and\n",
        "zoom the model. Examine it at large and small scales, and see whether\n",
        "you can spot patterns in the projected data and the clustering of\n",
        "labels.\n",
        "\n",
        "For better visibility, it’s recommended to:\n",
        "\n",
        "- Select “label” from the “Color by” drop-down on the left.\n",
        "- Toggle the Night Mode icon along the top to place the\n",
        "  light-colored images on a dark background.\n",
        "\n",
        "## Other Resources\n",
        "\n",
        "For more information, have a look at:\n",
        "\n",
        "- PyTorch documentation on [torch.utils.tensorboard.SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html?highlight=summarywriter)_\n",
        "- Tensorboard tutorial content in the [PyTorch.org Tutorials](https://pytorch.org/tutorials/)_ \n",
        "- For more information about TensorBoard, see the [TensorBoard\n",
        "  documentation](https://www.tensorflow.org/tensorboard)_\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.9"
    },
    "vscode": {
      "interpreter": {
        "hash": "cf18841ace8313d0bc088ca146c17a6c0040e82121d5cb75c0ea07172309253d"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
